Skip to content

refactor(estree/tokens): convert tokens spans before AST spans#19723

Merged
graphite-app[bot] merged 1 commit intomainfrom
om/02-22-refactor_estree_tokens_convert_tokens_spans_before_ast_spans
Feb 25, 2026
Merged

refactor(estree/tokens): convert tokens spans before AST spans#19723
graphite-app[bot] merged 1 commit intomainfrom
om/02-22-refactor_estree_tokens_convert_tokens_spans_before_ast_spans

Conversation

@overlookmotel
Copy link
Member

@overlookmotel overlookmotel commented Feb 25, 2026

Refactor.

Previously we converted spans from UTF-8 to UTF-16 in AST first, then serialized tokens. This meant that when serializing tokens, we had to convert spans to UTF-16 before comparing offsets of overrides to the token's start offset (because the override offsets are taken from the AST nodes during the AST walk pass).

Instead, convert token spans first, then the AST. This removes span conversion from the critical path. On a CPU with instruction-level parallelism, other work can start without being blocked on waiting for the result of span conversion, and the two can happen in parallel.

This should be a slight perf improvement, though it likely won't show up on CodSpeed benchmarks as CodSpeed's CPU simulation to basic to emulate instruction-level parallelism.

But mainly, the motivation is that this makes the process simpler and easier to reason about.

@github-actions github-actions bot added A-linter Area - Linter A-cli Area - CLI labels Feb 25, 2026
@github-actions github-actions bot added the C-cleanup Category - technical debt or refactoring. Solution not expected to change behavior label Feb 25, 2026
This was referenced Feb 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-cli Area - CLI A-linter Area - Linter A-linter-plugins Area - Linter JS plugins A-parser Area - Parser C-cleanup Category - technical debt or refactoring. Solution not expected to change behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants